The Digital Index: A World Below

We live in the age of indexical artificial intelligence. Data and algorithm combine to direct the flow of commodities and labor power, images and messages, through geographical and institutional space. Avant-garde AI algorithms called neural nets manipulate high volumes of data, forcing us to leave behind the sense that digital computation is abstract, disembodied, heuristic. This technology has become a core element of global capitalist infrastructure, as the parsing of data produced on digital platforms depends on these algorithms. Nets are currently used in targeted advertising, self-driving car technologies, pricing systems, social media, content-delivery platforms like Spotify and Netflix, facial recognition, medical imaging, judicial consulting—the list goes on.² The dangers for economic and racial justice have been widely demonstrated,³ but as we enter the petabyte era—or perhaps the exabyte or zettabyte era—it seems unlikely that the pace of adoption will slow.⁴

Computers were built to handle symbols assigned arbitrarily to arrangements of hardware with sets of switches operating in rigid syntax. We humans were supposed to be the channel through which the answers to queries flowed back into the world. The scientist at the terminal was supposed to ground the output and assign answers to contexts. But we have plugged our machines into each other and let them take on systemic autonomy, allowing algorithms to set price, determine financial speculation, and find matches between consumer and producer in online-to-offline services like rideshares. Mercedes Bunz has argued that nets “calculate meaning”: the signs that force our attention to our digital devices no longer seem a world apart, a source of consultation, but a world below—a semiotic infrastructure.⁵ Digital systems, relying on the neural net, have left the world of mere symbol behind and have begun to ground themselves here, now, for you—they are able to point to real states of affairs in a sign function known as the index.

We can neither afford to dismiss nor to accept fully the signification of digital systems. As we leave behind the sense that computers merely process symbols unattached from our physical and social realities, we run the risk of believing too much that machines really are intelligent. This is why data-driven has taken on the ring of necessity in our capitalist institutions. Indexical AI contrasts with the symbolic AI that dominated artificial intelligence research before 2000.⁶ And it uses this powerful referential function, the index, to gain the appearance of truth, the force of necessity. In the semiotic terms that we spell out below, nets indexicalize data to produce judgments about the world—judgments we take to be meaningful and to resemble our own judgments at our own risk. Neural nets disassemble images for recognition and production of new images, pointing us to what appear to be resemblances, images of something other than the pathways of the net itself. They also search for linguistic redundancy, indicating and replicating high-level generic qualities of natural language. Questions about how much the systems know about images or language, and through them the world, are missteps and should be abandoned in favor of concrete analysis of these now ubiquitous semiotic operators.

With the spread of these systems, the ability to dissent from their conclusions begins to fade, and the gap between their signification and ours closes. Subtending both the logistics of capital and the inequitable mangling of identity, nets have collapsed the space between infrastructure and representation, the small window of free air between immiseration and humiliation. Indexical AI seems to maintain the vector of history, the feeling of the necessity of capital, reinforcing “capitalist realism,” in Mark Fisher’s phrase.⁷ The sense that there is no alternative is the sense of being borne along on the “future flow” of history, as E. P. Thompson once put it, of simultaneous closure and progress, even if progress is along the darkest timeline. What McKenzie Wark calls “vectoralist” capitalism—or perhaps something “worse” than capitalism—is rooted in the literal indexical vectors of the neural net, which form a material and signifying grid in which we both live and think.⁸

When we accept the material and social reality established by these nets, we engage in what we call the naïve iconic interpretation of AI, an uncritical and ultimately metaphysical belief in the promise of intelligence. Analyzing digital semiotics allows us to peer inside what seem to be “black boxes,” revealing not an icon but an index.⁹ Critique of this new infrastructure must abandon the allied notions that digital systems occupy a merely logical space and that they know better than we do and should guide us.

Net Semiotics

A neural net is a mechanism that learns representations from data. Made up of nodes linked to one another with algorithmic instructions—in simplified form: first multiply, then sum; if less than 0, then delete—nets convene many simple parts to produce complex wholes. A net tasked with classifying canines first undergoes an extended period of training, during which it is exposed to as many relevant examples as is practical. Each image in the training set—an array of pixels plus a semantic label, recast as a single vector of information—is fed through the net, multiplied by random weights and a nonlinear function at each node, before finally arriving at the classification stage codified in the net’s output layer. Initially, the net will be highly prone to errors because the parameters that determine how nodes interact with one another begin in a randomized state. By using a technique for algorithmic optimization called stochastic gradient descent, however, the net can assign blame for these errors (or credit for precision) to individual neurons along the pathway from input to output. Over time, this process allows the net to refine the weighted parameters linking nodes gradually and to learn which features in the dataset are significant for detecting, for example, a Samoyed, even distinguishing it from a white wolf that often looks identical to the untrained human eye. The trained net possesses no holistic sense of Samoyedness but rather a complex architecture of indexical pathways that point to Samoyedness by capturing salient relations between features (fig. 1).

The semantic range of these algorithms relies on the second function of reference in Charles Sanders Peirce’s trichotomy of signs. For Peirce, there are three fundamental modes of designation of an object: the icon, the index, and the symbol. It is common say that the icon signifies by similarity with its object,¹⁰ the index by existentially affecting its object,¹¹ and the symbol by pure convention. The picture in the mind, the portrait and the algebraic equation are icons; the pointing finger, the relative pronoun, and the weathervane are indexes; and the letter, but also the data-point, is a symbol.¹² None of these operates in isolation, and in fact all three are at play in nearly every human use of signs. For example, Peirce identifies an “index of a peculiar kind” that is created by the tokens or instances of symbols in actual use.¹³ Because symbols express only generalities, their instances (words) must really affect other instances or signs in order to ground linguistic discourse. Digital machines, in order to ground their calculations, implement such a peculiar index.

Signs are composite, “indecomposable,” never reducible to a single function.¹⁴ Peirce writes that “a sign is anything which is related to a Second thing, its Object, in respect to a Quality, in such a way as to bring a Third thing, its Interpretant, into relation to the same Object.”¹⁵ The sign is a triadic relation between vehicle, object, and interpretant—it cannot be reduced to less than three.¹⁶ But the sign can communicate less than three—the “thirdness” of the sign communicates the “firstness” of a quality or a singularity in the icon and the “secondness” or relation in the index.¹⁷ Icons are “sources of discovery”: “a great distinguishing property of the icon is that by the direct observation of it other truths concerning its object can be discovered than those which suffice to determine its construction” (“I,” p. 279). Peirce gives the example of two photographs that together allow one to draw a map, converting the combined likeness of space into a diagram of territory. “Given a conventional or other general sign of an object,” he writes, “to deduce any other truth than that which it explicitly signifies, it is necessary, in all cases, to replace that sign by an icon” (“I,” p. 279). Even algebraic icons—equations—are such a source of discovery because they are made up of signs that communicate qualities, among which we select and compare. Selection is indexical because the index is a sign that communicates a relation (a “second”).¹⁸ Peirce pays special attention to the problem case of logical algebra in the Boolean tradition, which projects a dyadic world of indexes but fails to recognize its own status as a third, or sign-relation.¹⁹ Digital machines, built on an ultimately Boolean algebra that intentionally excludes triadic logic, extend this case and complicate it.²⁰ Indexical AI disproves the exclusion, pushing computing into a semantic domain by means of the peculiar, symbolic index, a third that communicates dyadic relations among unimaginably many data points. Early nets solved logical problems; current nets make judgments about creditworthiness, jail time, and other nondyadic things. The index in the neural net substitutes for and manipulates the other sign functions, heightening the sense that net output is meaningful, trustworthy, robust. Terrence Deacon argues that “only indexical relationships provide information,” while “iconic relationships” can be used to acquire information, and symbols constitute “relationships between forms of information.”²¹ The sheer complexity of this semiotic function, combined with the volume of data and amount of processing, causes us to lose sight of this information-pushing function, this indexical information.

Since Alan Turing threw down the gauntlet for machine intelligence, the question of imitation has been a backbone of AI research. Turing famously claimed that the way a machine could prove its intelligence would be by becoming indistinguishable from a human interlocutor in the imitation game. The theory of intelligence has always had this iconic root—because we do not know the mechanics of human intelligence (at least not in interaction with other humans), resemblance is the crucial feature of any AI.²² One can try to solve this problem of likeness by approximating the expression of intelligence or by imitating the underlying hardware, the brain. Both approaches take the surface feature of the icon—likeness—as the crucial element of AI. Machines must either be intelligent from similar physical resources to ours, or they must at least express themselves in signs that could only come from an intelligence like ours.

We call this implicitly iconic theory of intelligence the naïve iconic interpretation because the comparison rests on a resemblance unanchored in the expression of a first or quality.²³ The comparison of human and machine intelligence is slippery. The implicit icon (the quality intelligence) has no known domain—we are supposed to know it when we see it. And of course we do, which is why AI proceeds by provocative exhibitions—as the automata tradition did from antiquity on²⁴—from Ray Kurzweil’s machine-composed piano piece in 1965 to Lee Sedol’s defeat at the hands of the neural net AlphaGo in 2015.²⁵ These exhibitions always beg the question. The neural net might be like human intelligence in many different ways, but the naïve iconic interpretation is a metaphysical misstep. Neural nets operate indexically, as a quick glance at a canonical diagram of one suggests (fig. 2).²⁶

Data inputs are treated as indexes, fed forward and then backpropagated through the net.²⁷ Arrows—as pure an index as a diagram can feature—are ubiquitous in these visualizations. Nets are often characterized as patterndetectors, but they do not see patterns as a whole. At a depth of five to twenty hidden layers, the net can indeed distinguish Samoyeds from white wolves with striking accuracy, as Yann LeCun, Yoshua Bengio, and Geoffrey Hinton report.²⁸ But the comparison is limited to pointing. The job of the net is to form capillary connections between informational units and apply these connections after training. Whatever surplus in apparent intelligence it gains is given in the array of those units. When Bengio writes that “Deep learning allows the computer to build complex concepts out of simpler concepts,” with “each concept defined in relation to simpler concepts” and “more abstract representations computed in terms of less abstract ones,” the entire question of intelligence is begged.²⁹ The question of whether a net possesses a concept is being posed rather too early. The net finds its way to a pattern—Samoyed, including “not-white-wolf”—through iterative vector multiplication, creating a pathway that generalizes from object to class. Even the core metaphor, learning, renders an iconic interpretation of an indexical process. The metaphor suggests that this learning is like ours—or like intelligence as such—but does not specify how. The comparison buries the specificity that would help us understand these algorithms. Our proposal is that a version of Peirce’s peculiar index is at work here—that we have less an artificial intelligence than an automated index generator. Images and language, among other types of data, are disassembled, tokenized, and their iconic contents rearranged indexically. The output can be reintegrated into our collective knowledge, even given the status of an icon—everything has qualities, after all. But we should not accept the force of net-produced conclusions as though they were iconically grounded. The metaphor learning papers over the semiotic process, contributing to adoption but not critical analysis.³⁰

This is not to say that the net does not generate hypotheses. When Spotify’s algorithm expresses the determinate judgment “this is a grunge song,” we should strictly state this as “if grunge ranges over these songs, this specific song can be included.” Judgments of this sort are called “abduction.” François Chollet captures this ability of nets nicely.³¹ Rather than applying a general law to phenomena, or deriving a tentative law from phenomena, abductive reasoning is hypothetical; if there were a general law that included these examples, then this particular item would fall under it (fig. 3).³² The net finds the function rather than applying it. Luciana Parisi has argued that neural nets display just this kind of judgment, rendering them “transcendental[ly] instrumental”; they are tools that possess a sort of judgmental autonomy.³³ The icon drives abductive reasoning because we must identify and differentiate qualities by direct comparison to arrive at a hypothesis. But a neural net does not quite reason iconically, as we will argue.³⁴ It generates hypotheses in the form of indexical pathways that it traces between the tokens it gets as input. Pixels, for example, combine to make an image, which is tempting to call an icon (and may be interpreted as one). But the action of the net occurs on a disassembled icon, training on and producing an indexical pattern of tokens that can choose to construe as an icon once again—or not. In this respect, even the image fed through the neural net is more like a literary icon, a quality communicated in hierarchically arranged symbols, than an image understood as resembling its referent.³⁵ The naïve iconic interpretation of learning systems rests on this confusion. It takes the image as icon, the ability to identify images as iconic reasoning. Neural nets, however, are indexical all the way through.

Indexical AI

At the opposite extreme of the iconic interpretation of neural nets is the paradigm known as symbolic AI, which dominated AI research from its inception in the 1950s to the turn of the millennium. During this period, the paradigm that now encompasses deep learning went under many names—machine learning, parallel distributed processing, and connectionism.³⁶ All have shared the conception of the neural net, and all to some extent are “inspired by the actual biology of the brain,”³⁷ with neuroscientists and AI researchers working interdisciplinarily, sharing an epistemology opposed to a “symbolic” construal of intelligence.³⁸

In their 1975 Turing Award speech, “Computer Science as Empirical Inquiry”—the canonical explanation of the symbolic AI program—Herbert Simon and Alan Newell write that a “physical symbol system consists of a set of entities, called symbols, which are physical patterns that can occur as components of another type of entity called an expression (or symbol structure).”³⁹ These patterns are able to designate or interpret objects belonging to the system’s domain. The project of building digital computers in the first place was partly the construction of artificial languages in which hardware could operate, to carry out operations on command, answering queries in numbers or language. The symbol system is unequivocal and arbitrary in its designation of values and hierarchically coded, in keeping with Peirce’s definition of the symbol, or Ferdinand de Saussure’s famous “arbitrariness of the sign.”⁴⁰ Preloaded with meaning by imputation, these systems sever the connection between semantic unit and reference, gaining power at the syntactic level purchased at the price of worldly knowledge.

The connectionists thought the symbolic picture was incomplete and sometimes spoke of a “subsymbolic paradigm.”⁴¹ Symbols have semantic content arbitrarily but unequivocally associated with them. But what if intelligence is made up not of symbols but of connections between semantic units? The suspicion that the brain’s neurons generate mental content through weighted links led to this alternate proposal.⁴² What is underneath the symbols in the “subsymbolic paradigm” is also between the symbols—links, connections: in a word, indexes. Nonsymbolic AI, then, is always a form of indexical AI.⁴³

There is a critical tradition in the history of AI that has sometimes conceived of intelligence as indexical. Its main proponent is Philip Agre, who places himself in the phenomenological tradition of AI critique that began with Hubert Dreyfus’s Heideggerean attack on symbolic AI in the late 1960s.⁴⁴ Arguing that “indexicality has been almost entirely absent from AI research,” Agre proposed a “deictic ontology”—achieving contextual awareness by physical indexes—which “can be defined only in indexical and functional terms, that is, in relation to an agent’s spatial location, social position, or current or typical goals or projects.”⁴⁵ This tradition conceives of intelligence as indexical and so makes indexicality an aspiration of an alternate AI. But this aspiration has blinded the tradition, which still seeks to realize “intelligence,” to the indexical effects of actually existing AI. Take, for example, Brian Cantwell Smith’s recent argument that the crucial concept for deep learning is context:

The Shape of a Neural Net

The net is a complex function with a concrete shape. In training, the net gains the ability to point at features, and composite sets of features add up to objects. Because the output is then either a determinate judgment (“this is a Samoyed, not a white wolf”) or an image or text that resembles human pictorial or linguistic content, we are predisposed to think of that output as iconic. But the net does not operate iconically; it treats the iconic aspects of its input indexically. Its rhetorical power and semiotic limitation come from its shape.

What has become the neural net was first proposed as a “nervous net” by cybernetician Warren McCulloch and his prodigy student Walter Pitts.⁴⁸ They sketched diagrams of formalized neurons that would allow either a brain or a machine (not yet a computer in 1943) to perform Boolean operations.⁴⁹ McCulloch and Pitts thought that a weight-conditional set of neurons, if complex enough, might be able to achieve true knowledge of the world (fig. 4).⁵⁰

The diagram (c) shows an AND function: if (1) and (2) are both tripped across the synapse, (3) fires, but not if only one or the other does. Many early machine learning systems were built on this insight, including Frank Rosenblatt’s perceptron.⁵¹ But these machines were unable to perform an XOR function, in which the neuron fires if one or the other of its inputs trips, but not both. It was this problem that proved the efficacy of deep nets, nets with hidden layers, meaning the calculations at these layers are inaccessible to the input and output stations.

A single hidden unit turned out to be enough to solve the XOR problem. As David Rumelhart, Geoffrey Hinton, and Ronald Williams were able to show, a hidden unit allowed the net to learn weights that rarely failed to execute XOR.⁵² The middle layer creates a step between input and output that gives the calculations shape—this is why nets are described as having architecture. This early diagram visualizes the flow of calculations labeled with the learned parameters to solve XOR (fig. 5). If we look at the truth tables for inclusive or (OR) and XOR, we can see in semiotic terms why a hidden layer is needed to solve this equation (fig. 6).

It is tempting to think of these tables as pure binary symbolic expressions, as they contain only logical tokens with binary values. But equations—even when they are not laid out in tables—are icons displaying the reasoning they denote (fig. 7).⁵³ They are signs that communicate a “first,” in this case the quality of a judgment or style of reasoning, and therefore “sources of discovery.” But while OR operates on one binary logic (the presence of 1 as input means 1 as output), XOR shifts to another, in which two trues become a false, basing the value 1 on the presence of identical or different inputs. This requires an extra step in reasoning, a pathway through the icon that dissociates binary arithmetic from the query. It is this extra step that the hidden layer allows for. As Rumelhart and his colleagues noted, full isolation of input from output required even more layers and led to even more general function finding.

If XOR is an algebraic icon, then we can see that the net does not possess that icon as input. Instead, it establishes an indexical pattern that generalizes to instances of the function. In the case of a Boolean function, if the output is correct, it must be a matter of indifference whether the algorithmic architecture is iconic or indexical. But this indifference will not survive the introduction of petabytes of social data. Early nets were able to solve logical problems without having the icon as a source of discovery, a way of ensuring that truths learned from the data actually held. Logic itself made those assurances in problems like these. But if the source of discovery for data about creditworthiness, racial identity, or market fluctuations is not iconic, no a priori set of rules comes to our aid in reading the indexical results.

When humans input icons into deep learning systems, asking them to produce real answers to real questions, these systems indexicalize the icons they are asked to handle. After the XOR problem, Rumelhart, Hinton, and Williams introduce symmetry as a problem that only hidden layers can solve. Symmetry cannot be detected in the comparison of sums: “an individual input unit, considered alone, provides no evidence about the symmetry or non-symmetry of the whole input vector, so simply adding up the evidence from the individual input units is insufficient.”⁵⁴ Finding these particular balance points is the work of the “intermediate layers.”⁵⁵

Symmetry is the concept of internal iconicity. A symmetrical object resembles itself across an axis. Even more obviously than in the case of the XOR function, the detection of symmetry requires the net to trace two lines of equivalency. When you or I look at the figure, we can detect symmetry by scanning the parallel numbers—a slightly different operation but with the same result, as when I see an apple as symmetrical (fig. 8). The net, however, cannot rely on the comparison of arbitrary points against the whole to check symmetry. At no point in the forward or backward passes can the net compare the whole shape to an individual point (this is the meaning of hidden layers). Looked at from the outside, the pathways that the net establishes make up an icon. But no one—nothing—looks at them from the outside. The reason the icon is a source of the discovery of truth not contained in the rules of its construction is that a human interpretant of an icon can compare it to its referent. But the source of discovery in the neural net is not the icon as such. Any truth the net discovers must be a combination of the symbolic input and the indexical pathways that the net establishes—the source of discovery is the pattern through the layers: indexical tokens arranged by backpropagation.⁵⁶ This peculiar sign function is the digital index.

Once there are hidden layers the system cannot learn merely by comparing input and output. This is because the output alone gives no sense of where the algorithm’s weights should be adjusted to optimize the outcome, known as the “credit (or blame) assignment problem” (D, p. 186). The technique that solved this problem—which is regarded as the crucial step to the spread of nets around the turn of the millennium—is the backpropagation algorithm.⁵⁷ As John Kelleher points out, the entire process of correcting error in a net is often referred to as “backpropagation,” a two-step process—measuring the error, updating the weights (D, p. 209). Once the net has performed the “forward pass” (multiplication and sums of the layers), there follows the “backward pass,” which assigns credit and blame (an “error value,” ∂) to each node.⁵⁸ Although these values are summed, the layers are not visible to the output or the input, with the result that the net takes the shape of the indexical patterns it establishes between the layers in the passes.⁵⁹ The pattern includes the input, output, and individual points, but the unity of this pattern—the shape of a neural net—is not available to the net and in a sense also does not exist for us, as we cannot make sense of its extremely high volume of data and processing at each neuron. The pattern exists only indexically, as the throughput of the algorithm. In distinguishing dog from wolf, the net is not comparing dogs and wolves. It is pointing by means of a distributed series of indexical tokens at the label Samoyed, and it does this by means of the digital, or algorithmic, index.

Indexing Images

The digital index is a semiotic operation that drives net-based image processing, which might otherwise appear to be an iconic operation. The images in figure 9 were produced by a generative adversarial neural net (GAN) designed by engineers at NVIDIA, in which two nets are place in competition with one another, one finding flaws in the other’s output (fig. 9). The faces in this image do not belong to any humans. This sort of output is hard to resist taking as a source of discovery—surely even if we avoid being tricked by these images, we must think they have captured some quality like faciality. But the icon face is produced, in this case as in all nets, indexically.

This GAN relies on convolutional neural nets (CNN), which have proven particularly well-equipped for classification and regression tasks involving the sensory domains of vision and voice, recognizing the faces in your Facebook uploads or parsing your commands to Siri. These use cases tend to involve inputs organized into an array structure, whether two-dimensional bitmaps or one-dimensional time-sequence data. Like all contemporary neural networks, CNNs operate with backpropagation but draw much of their interpretive power from the interplay of distinctive architectural elements. This architecture has proven to be generative precisely inasmuch as it is capable of achieving feats of emergence that are ostensibly incompatible with the digital medium.⁶⁰ CNNs demonstrate that digital information is hardly limited to the symbolic register, and computers are anything but mere symbol manipulators. Yet they also are not in the business of processing icons, however much we may tend to think of image and icon as adjacent. CNNs indexicalize images, including their iconic aspects, and either class them or generate proximate images by means of the digital index.

The architectural elements of the CNN responsible for the indexicalization of images were laid out in 1980 by Kunihiko Fukushima.⁶¹ He called his framework the “neocognitron,” and it consisted of a multilayered structure of “simple” and “complex” cells.⁶² Inspired by the landmark studies undertaken by neurophysiologists David Hubel and Torsten Wiesel that discovered a hierarchically structured neural network in the feline visual cortex, the neocognitron architecture consisted of alternating layers of S-cells and C-cells (fig. 10).⁶³ S-cells distinguish excitatory from inhibitory regions, working like a flashlight scanning a dark interior, illuminating discrete parts of the uncertain whole. The S-cell performs something like the gestalt operation of interleaving foreground and background, bringing the two into relation with one another. The C-cell takes as its input an amalgamation of the bounded regions captured by the preceding S-cells and congeals them into a higher-order output. Because a single C-cell effectively transforms the features captured by different S-cells, the simple-to-complex data exchange between these layers renders an output that is comparatively agnostic to the spatial (or temporal) contiguities that the S-cell is designed to respond to. S-cells want to know where and when certain features exist, while C-cells only need to know that these features exist. The neocognitron adumbrated the crucial feature of contemporary CNNs: S-cells burned indexes through the data, while C-cells pooled those indexes, creating an indexical sum to identify an object in the image.

Beginning in the late 1980s, LeCun, working with a rotating slate of collaborators, integrated the promising new technique of backpropagation into the convolutional architecture,⁶⁴ developing LeNet to recognize handwritten letters.⁶⁵ LeNet’s convolutional layers produce indexical feature maps, and pooling (or subsampling) layers reassemble these features into potential icons. Finally, for the net to render a judgment interpretable to humans, a fully connected output layer converts the input from the final pooling layer into another index, flattening the multidimensional array into something resembling a traditional look-up table (not dissimilar from a book’s index). This duo of convolution and pooling enables the net to realize qualitative novelty out of a quantitative substrate. CNNs, ultimately, are two-part navigational devices: convolutional layers carve new informational pathways through uncharted latent space, and pooling layers groom these pathways to ensure easier transit.

Consider a digital image: twenty-eight pixels tall, twenty-eight pixels wide (fig. 11). This array of 784 pixels presents to the human eye a grayscale raster depicting a handwritten digit. It is one of 9,298 images provided to LeCun’s team by the US Postal Service, culled and normalized from a database of zip codes. Because the image is black-and-white, the input layer has the dimensions 28 × 28 × 1, and the output layer is 10 × 1, each neuron providing a weighted probability that the input contains one of the ten possible digits.⁶⁶ Four layers in LeNet play alternating roles: H1 and H3 are convolutional layers, while H2 and H4 are subsampling layers. Convolutional layers, in essence, add depth: H1 puffs up the flat, two-dimensional input array into a stack of four arrays, each dedicated to a specific feature detection task. The subsampling operation performed by H2, in turn, retains the fourfold depth model produced by H1 but reduces the breadth of each layer by half. H3 and H4 continue this process: the former expands the input received from H2, tripling its depth from four feature-detection planes to twelve, and sends this narrow-but-deep data to H4, where it is narrowed by half once again.⁶⁷

Convolution is a mathematical operation that produces a third function out of two input functions. The first function is the input, the second function is the kernel (a matrix that passes over the data image, checking for the desired features), and the third function is the feature map. The feature map is a function that represents the influence of the kernel on the input, and it is here that we see the net indexicalize the image, in a process that resembles scanning. H1 produces a feature map comprising new signifying elements that represent the influence of convolving the values in a 5 × 5 kernel over the values in each possible 5 × 5 region of the input. The indexical tensor that this operation produces has the dimensions 24 × 24 × 4, the latter value indicating the number of unique filters that have convolved the input data. The four feature maps in LeNet’s H1 layer depict what the net has determined to be salient for recognizing the digit 0. These features serve as heuristics for discovering granular primitives like edges, curves, and contours.

On first glance, H2 seems to do similar work to H1. But while both operate on local patches of their input, with H2 employing a smaller kernel than its predecessor, the two layers differ markedly in how they treat this data. Unlike the convolutional filter, which moves across the input in overlapping strides, the receptive fields in the subsampling layer do not overlap. Likewise, they perform a more elementary bit of math than H1, by simply taking a local average of the values in each receptive field. This averaging function is inelegant compared to the dynamic processing of convolution, but its aim is to preserve as much of the indexical information elicited by the preceding layer, while maximizing the net’s efficiency by reducing the overall breadth. The tensor produced at this stage retains the layers it received as input from the convolutional layer, but the resolution of each layer has been halved, producing a new 12 × 12 × 4 mapping of the data’s features. The representations produced by pooling trade resolution for resilience, ensuring like features can be detected as like, despite the innumerable contingencies in real-world input data. Pooling endows the net with translational invariance allowing it to redeploy the features learned in the convolutional stage with more accuracy and efficiency. Pooling kernels essentially force the net to treat inputs with slight variance as like one another. Unlike the indexicalization performed by the convolutional layer, pooling layers like H2 determine which of the differences in the input data can be ignored, and which differences are salient. By instructing the net not to notice certain differences, H2 exercises a thresholding operation to produce a map of potential icons, which will be shuttled onward to H3 for further indexicalization. Recall Deacon’s argument, which captures this process: “Only indexical relationships directly provide information … iconic relationships [present] the possibility of being used to acquire information.”⁶⁸ The CNN architecture is designed to maximize potential iconicity by stacking convolutional and pooling layers in the indexical pathway (fig. 12).⁶⁹ H3 seizes on this iconic potentiality and produces a third feature map with even greater depth. H4 retains this newfound depth, while once again halving the breadth producing a final tensor with the dimensions 4 × 4 × 12. Each of the 192 neurons in this final pooling layer are connected to the output layer of LeNet, which bears ten total neurons, one for each possible digit. Unlike the generative indexing performed by convolutional layers, this fully connected layer compresses the feature map of H4 into a rigid table of reference, yoked to a set of predetermined tokens—another index.

The CNN made headlines in 2012 by taking top prize at an annual computer-vision competition.⁷⁰ It was not the architecture alone but the unprecedented scale of the winning net—colloquially deemed AlexNet, after its creator—that was noteworthy. Boasting eight layers, five of which were convolutional, and more than sixty million parameters spread across nested pooling layers and the fully connected output, the network’s performance cemented the new paradigm of deep learning. As these indexical algorithms have produced increasingly iconic images, indexical AI has spread, its output ever harder to doubt.

Indexing Language

Nets processing natural language have gained the ability to produce linguistic icons. Operating indexically like all nets, and discovering the heavily indexical aspects of language, these systems have nevertheless demonstrated an ability to produce something like literary language, expression-conveying qualities not found in their parts but only in a holistic fashion. This nascent literary quality might be even more responsible than the achievements of CNNs for our tendency to trust these systems. Recurrent neural nets (RNNs) and their successor, the Transformer Architecture, search for patterns in temporalized sequences. Where the CNN indexicalizes and reassembles an icon, RNNs and the Transformer Architecture, which process natural language, produce a potential icon out of a combination of symbols and indexes.

Net-based natural language processing has become “unreasonab[ly] effective,” to modify a phrase that engineers Peter Norvig and Andrej Karpathy separately borrowed.⁷¹ Mathematician Eugene Wigener had claimed that mathematics was unreasonably able to explain empirical phenomena, giving us the sense that we “‘got something out’” of the equations that we did not put in.⁷² Mathematics, in other words, is a source of discovery, and “unreasonable effectiveness” is a hyperbolic statement of the naïve iconic interpretation of neural nets.⁷³

In order to process language (or other sequential data), nets must build an extra set of feedback loops into each calculation layer. Whether a pixel is red or not might be highly dependent on the next pixel over but is usually not dependent on a pixel on the other side of the image. Language, however, displays the opposite tendency. In addition to some short-range dependencies (“Mary runs”) language has long-range dependencies (“Mary, even though she doesn’t want to, runs to maintain fitness”). The conjugation of a verb is a classical indexical component in a sentence, as the third-person singular s distinguishes Mary from other possible subjects. These nets have gone one step further and solved some Winograd schema problems, in which the sense of a sentence is dependent on “knowledge of the world.”⁷⁴ Terry Winograd’s first example of this sort of problem was the sentence “The city councilmen refused the demonstrators a permit because they advocated revolution.”⁷⁵ “They” in this sentence is what Roman Jakobson called a “shifter,” in which meaning “cannot be defined without a reference to the message.”⁷⁶ A shifter is an indexical function that conjoins a symbol and an icon (the singular state of affairs “this,” for example). Nets can reduce semantic ambiguity of this kind, linking “they” with the correct referent. But they exploit this capacity to generate much higher-level indexes, ones that connect not only pronouns but also larger units of meaning like genres and styles. Where CNNs indexicalize an icon, nets processing language indexically isolate and generate linguistic icons—meaning in use—as we will see.

All nets processing language use some form of embedding to make strings of words into data input. Algorithms like word2vec embed words by assigning them to tokens (vectors, in this case) to input into a net. Initial vectors are random, and the algorithm “learns” that “words that appear in similar contexts have similar meanings” (D, p. 181). These vectorized words then become the input for neural nets, which concatenate these meanings into syntactically correct, semantically approximate expressions. Tomas Mikolov and his colleagues express surprise at the ability of their word-vectors to learn analogy: “The result of a vector calculation vec(‘Madrid’) - vec(‘Spain’) + vec(‘France’) is closer to vec(‘Paris’) than to any other word vector.”⁷⁷ These embedding algorithms have managed to produce the prediction “queen” in answer to the query: “king – man + woman.” The autonomy of such a semantic space resembles what Jakobson called “contiguity disorder,” a type of aphasia characterized by “agrammatism.”⁷⁸ For Jakobson, this disorder means that the patient possesses metalanguage but no object language—they can substitute meanings for one another but cannot synthesize sentences that rely on context. They understand “gestalt” but are trapped in its paradigm.⁷⁹ As in word2vec, the metaphoric pole of language becomes a series of proximities without syntax. Word embedding produces this semantic space, and the net is left to learn the rules of contiguity.

The RNN functions on a simple premise: at each layer, the output must contain both the normal layer forwarding (activation, weights, input values, and bias) and a hidden state that carries information forward from earlier in the layers—context, in the form of a vector storing previous information in the sequence. The recurrence is a for loop, in programming terms, iterating through the string until all relevant dependencies are learned. But the memory buffer will slowly minimize lower probabilities out of existence, a problem called the vanishing gradient. The addition of “long short-term memory” was meant to allow RNNs to index longer-range dependencies.⁸⁰ This architecture differentiates the memory buffer by making it into a cell composed of a forget gate that pushes some values near 0 and not does not pass them forward (a mathematical index).⁸¹ The LSTM generates a kind of indexical heatmap of a text, a linguistic version of the feature maps produced by convolutional layers. But the LSTM can only reduce, not eliminate, the vanishing gradient of the RNN, leading to their replacement, in 2017, by the “Transformer architecture,” which relies on “self-attention” to relate “different positions of a single sequence in order to compute a representation of the sequence.”⁸² Rather than storing information through the forward pass, attention selects for relevant information at each node, imitating the indexical function of consciousness.

Paying attention is patently indexical. Think of the ubiquitous banner messages that force our eyes to our screens; these are indexes driving the attention economy.⁸³ When we pay attention, we choose an anchor for our representations and allow the mind to range over qualities, relations, and other signs around that center. Peirce notes that the “subject of a proposition” is an attentional index, in this sense, communicating a “quality” against which the predicate can be checked.⁸⁴ “Self-attention” performs just this operation.

The Transformer contains three separate attention mechanisms each with multiple attentions functions (multihead): one for its encoder, one for its decoder, and one for their interaction called the “encoder-decoder” (figs. 13–14).⁸⁵

Let’s say you have a query: What should the next word be in a sentence? What is the correct word at a given position for a translation of a sentence from French to English? Attention takes two input vectors—values and keys, where keys are indexed to the values, such that they are often initially equivalent—and multiplies the query vector by the specific key, which produces a set of scores for the k vector, establishing “how much each word will be expressed at this position” (fig. 15).⁸⁶ For technical reasons, all these products will fall between 0 and 1. The attention head then uses an exponentialization function called the “softmax,” which creates two groups of the product values, one very close to 0 and one very close to 1 (fig. 16).⁸⁷ The softmax mathematically indexes a set of relevant units. These products are then multiplied by the values vectors and summed to produce the result of this attention. Once all three attention heads have acted, a prediction is created on the basis of a composite sense of relevance. The multihead approach, which launches the attention calculation into very high dimensions, has allowed the Transformer to learn both semantic space and syntactic order.

The old suspicion that digital computers are syntax machines with no capacity for meaning should finally be laid to rest. These machines do not suffer from Jakobson’s other aphasic pathology, “similarity disorder,” never knowing content but only order. Jakobson writes that patients lacking metaphor and possessing only metonymy “fail to shift, as Peirce would say, from an index or icon to a corresponding verbal symbol.”⁸⁸ In other words, they fail to attend to the shifts in symbol-icon connections, a form of semantic creep.⁸⁹ The Attention mechanism uses the softmax index to find patterns among the symbols that correspond to redundancies in usage that we find meaningful or construe as icons.

Language models based on the Transformer architecture have been able to generate philosophical essays about themselves, consistent if imperfect poetry in various author styles, and even to write code (or what looks like code) on command.⁹⁰ These large-scale models create composite indexes based on high-dimensional multihead attention mechanisms that trace the distance between very complex qualities in language, allowing them to communicate firsts or present us with potential icons.⁹¹ At scale, these potential icons seem to carry their own rhetorical force, pushing adoption of the models in spite of their risks.⁹²

Take the celebrated article one such model produced (fig. 17).⁹³ On a premise similar to that of Arthur Conan Doyle’s The Lost World (1912), the algorithm was able to generate a formally correct news article, respecting the generic requirements with only minor flaws (dialect or dialectic). Successes like this have led to debates about these systems has focused on whether they have become intelligent by learning contextual or worldly information from these very large language datasets, a notion that falls squarely into the naïve iconic interpretation of AI, failing to connect architecture and expression in concrete analysis.⁹⁴

When the article attributes four horns to the eponymously single-horned creatures, the error is an index that dislocates us from the universe of discourse in which the article otherwise immerses us. Whether science fictional, fantastical, or as narrow as binary logic, universe of discourse require “an index in the environment common to speaker and auditor.”⁹⁵ This index—computer science calls it “\interoperability, and structuralism calls it the phatic function—must exist because universes “cannot be described.”⁹⁶ To speak of unicorns is to locate ourselves in a universe we fall out of if they are ascribed four horns. Semantic entities like unicorns are not grounded only in the mental picture of a flying horned horse. They possess, as part of their meaning, an indication of potential universes to which they belong, a kind of semantic-indexical embedding. The unicorns article barely fails to immerse us in such a universe (indeed, we have never seen the contradiction of the horns pointed out in the breathless literature that followed on this article).

Large-scale language models arrive at the synthesis of text indexically, as they rely on the attention mechanism. But they can communicate a first, a quality—in this case, genre. What impresses the reader of the article is that it is a news article, that it appears as science reporting. In other words, the attention mechanism drawn out over a very large corpus is able to produce generically valid expressions, an instance of something we immediately recognize but which was produced by billions of indexical parameters operating on trillions of symbol instances. The naïve iconic interpretation comes from this feeling of recognition, taking the quality as the sign. Large-scale language models generate powerful linguistic icons from high-dimensional digital indexes. These icons are not just orderings of words that resemble the flow of thought.⁹⁷ They are closer to W. J. T. Mitchell’s account of the “‘verbal icon,’” communicating singular linguistic qualities, not limited to verbal pictures, but also including genre and mood.⁹⁸ To the extent that neural nets can generate such qualities, they participate in the literary aspect—what Jakobson called the “poetic function”—of language, where language’s object domain is language itself.⁹⁹ This reflexive appearance underwrites both the adoption of these systems and the naïve iconic interpretation of their abilities. We are in the process, not yet complete, of plugging these powerful index machines into our social-conceptual icons, allowing their expressions to adopt the ring of truth, the benefit of the doubt. Indexical AI lacks icons of its own, but it can generate signs we are prone to take as sources of discovery.

Indexical Labor and Algorithmic Capital

Indexical AI has become a “general condition of production,” linking the moving parts of the economy to each other.¹⁰⁰ The platform economy is not limited to targeted ads on social media, as important at these are for the indexing of consumer desires and the alteration of production. We drive using nets, taking unfamiliar routes dictated by Google data. The prices we see on Amazon, which increasingly pressure offline prices too, are probably the result of personalized net-based fluctuations.¹⁰¹ When you ask a rideshare to take you to the airport, the matching algorithm points driver and rider to each other and points them through space to a semantified goal (such as “terminal C”). When we follow Google’s routes, follow through on an Amazon recommendation, or confirm a time and location for car pickup, we enrich these companies’ indices. We even add data capital to these companies when we deviate from their routes, decline their recommendations, or cancel a pickup. More data means more flexible and powerful algorithmic indexes. This semiotic infrastructure bleeds upward into the daily interpretive tasks we face in the digital world. The digital index manipulates and shadows identity,¹⁰² while subsuming labor in the platform economy.¹⁰³ AI links our daily hermeneutic activity to global logistics by means of the index, tightening the loop of desire and understanding with production and distribution.¹⁰⁴ The naïve iconic interpretation of the net is virtually forced on us—resistance to hype tends to be futile.

Underlying this semiotics, and the daily hermeneutics it forces on us, is a reordering of the elements of capital. What appears to be a technology of distribution now shapes both labor conditions and capital itself. The index that appears to us as a point, a context, or a location is the channel along which labor flows. The partly automated data flow between those channels and the allocation of the resources of enterprise make up the structure of contemporary capital, an attractive force organizing the chaos of digital signs. We formulate this conclusion as two theses for future work: (1) indexical AI valorizes data by subsuming labor into the algorithmic index of capital; (2) the shape of capital is a feedback loop, partially automated by indexical AI, among context and action, data and judgment.

Hito Steyerl has identified the condition of ubiquitous ambient digital semiotics as “free fall,” in which grounding is only possible as if from above, so that data panoramas “do not actually portray a stable ground.”¹⁰⁵ But even if free fall continues indefinitely, it must be falling for the metaphor to work. The gravity in digital free fall is the index specific to contemporary AI. As we enter the age of ever bigger data, learning systems have yawned into existence as mega indexes tracing the outline of capital today.